Image capture with context data overlay

ABSTRACT

Techniques are described for capturing an image of a document or other type of physical object, and presenting the image in a user interface (UI) with an overlay that includes context information regarding the document or other physical object in the image. An application running on a portable computing device receives an image of the document that is captured using a camera of the device. The application may perform an initial analysis to identify one or more data elements present in the document, such as certain words, phrases, paragraphs, and so forth. The data elements can be uploaded to a remote service that analyzes the data elements and returns context data which is presented as an overlay to the image of the document. The context data can then be presented in an overlay to the presented image of the physical object.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of and claims priorityunder 35 U.S.C. § 120 to U.S. application Ser. No. 16/212,006, filed onDec. 6, 2018, which is related to, and claims benefit of, U.S.Provisional Patent Application Ser. No. 62/596,042, titled “ImageCapture With Context Data Overlay,” which was filed on Dec. 7, 2017, theentire contents of each of which are incorporated herein by reference.

BACKGROUND

In the course of their daily lives, individuals may be required to reada variety of documents related to various topics and with varyingdegrees of complexity. Some such documents may include language that isdifficult to understand for the average reader who may lack expertise inthe subject matter of the document. For example, making a high valuepurchase, such as a home purchase, can be stressful for manyindividuals. Applying for and obtaining a mortgage to purchase a home isa complex process, and may be especially overwhelming for first timehomebuyers. Individuals may see a property that they like, but may notknow how to find a reliable real estate agent, get prequalified for amortgage, determine what amount of mortgage they need, and/or determinewhich products and terms are suitable. Moreover, during the process,individuals generally need to handle many documents of different types,which can be difficult to understand. After a loan application goes intoprocessing, the process can become even more confusing and it may bedifficult for individuals to obtain status information regarding theirapplication.

SUMMARY

Implementations of the present disclosure are generally directed toimage capture and presentation with context data overlay. Morespecifically, implementations are directed to presenting a capturedimage of a document or other physical object in a user interface, with apresented overlay that includes context data regarding the document orother physical object being shown.

In general, innovative aspects of the subject matter described in thisspecification can be embodied in methods that include operations of:receiving at least one image of a physical document, the at least oneimage captured by a camera of the portable computing device; analyzingthe at least one image to identify at least one data element of thephysical document; communicating, over at least one network to at leastone server device, the at least one data element and receiving, from theat least one server device, context data associated with the at leastone data element, the context data communicated from the at least oneserver device over the at least one network; and displaying the at leastone image with an overlay that includes at least a portion of thecontext data.

Implementations can optionally include one or more of the followingfeatures: the at least one server device executes a service thatdetermines the context data based on the at least one data element; thecontext data includes a control that is displayed in the overlay andthat is selectable to display additional information describing the atleast one data element; the context data includes a control that isdisplayed in the overlay and that is selectable to launch a video chatsession between a user of the portable computing device and a servicerepresentative; the context data includes one or more of text data,video data, and another image; the at least one data element includes aloan identifier; the context data includes information describing astatus of a loan application corresponding to the loan identifier; theat least one data element is a property identifier; and/or the contextdata includes information describing a property corresponding to theproperty identifier.

Other implementations of any of the above aspects include correspondingsystems, apparatus, and computer programs that are configured to performthe actions of the methods, encoded on computer storage devices. Thepresent disclosure also provides a computer-readable storage mediumcoupled to one or more processors and having instructions stored thereonwhich, when executed by the one or more processors, cause the one ormore processors to perform operations in accordance with implementationsof the methods provided herein. The present disclosure further providesa system for implementing the methods provided herein. The systemincludes one or more processors, and a computer-readable storage mediumcoupled to the one or more processors having instructions stored thereonwhich, when executed by the one or more processors, cause the one ormore processors to perform operations in accordance with implementationsof the methods provided herein.

Implementations of the present disclosure provide one or more of thefollowing technical advantages and improvements over traditionalsystems. Implementations provide a context overlay that is presentedwith image(s) of a document or other imaged physical object in the userinterface of an application. The context overlay includes context datathat describes the imaged document or other object. Accordingly, bypresenting context data in a ready-to-consume format along with theobject being examined, implementations avoid the need for additionalnetwork requests or other processing to retrieve and present the contextinformation, thus making more efficient use of network bandwidth,processing power, storage capacity, active memory, and/or othercomputing resources compared to traditional, previously availablesolutions. Implementations also provide an overlay within a userinterface that allows users to avoid making numerous clicks and/ornavigation steps to see the context data, given that various options toaccess help from a service representative, initiate loan applications,and/or perform other actions are presented as part of the context datain the overlay. The avoidance of extra user interface actions alsoavoids the expenditure of computing resources, such as those listedabove. The context data can include cached and/or dynamic data elements,as appropriate. Dynamic data elements provide real time information.

It is appreciated that aspects and features in accordance with thepresent disclosure can include any combination of the aspects andfeatures described herein. That is, aspects and features in accordancewith the present disclosure are not limited to the combinations ofaspects and features specifically described herein, but also include anycombination of the aspects and features provided.

The details of one or more implementations of the present disclosure areset forth in the accompanying drawings and the description below. Otherfeatures and advantages of the present disclosure will be apparent fromthe description and drawings, and from the claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 depicts an example system for presenting a document image with acontext data overlay, according to implementations of the presentdisclosure.

FIG. 2 depicts an example system for presenting an object image with acontext data overlay, according to implementations of the presentdisclosure.

FIG. 3 depicts a flow diagram of an example process for presenting adocument image with a context data overlay, according to implementationsof the present disclosure.

FIG. 4 depicts a flow diagram of an example process for presenting anobject image with a context data overlay, according to implementationsof the present disclosure.

FIG. 5 depicts an example computing system, according to implementationsof the present disclosure.

DETAILED DESCRIPTION

Implementations of the present disclosure are directed to systems,devices, methods, and computer-readable media for capturing an image ofa document or other type of physical object, and presenting the image ina user interface (UI) with an overlay that includes context informationregarding the document or other physical object in the image. In someimplementations, the object is a physical document (e.g., a printeddocument). An application running on a portable computing device (e.g.,smartphone, tablet computer, etc.) receives an image of the documentthat is captured using a camera of the device. The application mayperform an initial analysis to identify one or more data elementspresent in the document, such as certain words, phrases, paragraphs, andso forth. The data elements can be uploaded to a remote service thatanalyzes the data elements and returns context data which is presentedas an overlay to the image of the document. In some implementations, theobject is some other type of physical object, such as a house or otherpiece of real property. The image of the object is captured and uploadedto the remote service with location data describing the location of theportable computing device that was used to capture the image. The remoteservice can analyze the image and the location data to generate contextdata regarding the physical object. The context data can then becommunicated to the portable computing device for presentation in anoverlay to the image of the physical object.

FIG. 1 depicts an example system for presenting a document image with acontext data overlay, according to implementations of the presentdisclosure. A user device 102 executes an application 104.Implementations support the use of any suitable type of user device 102that a user 102 can use to interact with the application 104. In someimplementations, the application 104 may be an augmented reality and/orvirtual reality application that executes on a computing device thatprovides an augmented reality and/or virtual reality interface.Alternatively, the user device 102 can execute an application 104 thatis any other suitable type of application, such as a web applicationexecuting in a browser or other container for web content, or a mobileapplication (app) configured to execute on the user device 102. In someimplementations, the user device 102 is a portable computing device suchas a smartphone, tablet computer, wearable computer, and so forth. Theuser device 102 can also be a less readily portable type of computingdevice, such as a desktop computer, laptop computer, smart appliance,gaming console, and so forth.

The user device 102 can include one or more cameras that captureimage(s) and/or video data of a field of view in proximity to the userdevice 102. In some instances, the camera(s) 106 may be peripheraldevice(s) that are connected to the user device 102 over a wired orwireless network, and are not built-in components of the user device102. The camera(s) 106 can capture one or more image(s), and/or videodata, of a physical object. In the example of FIG. 1, the imaged objectis a physical document 108, such as a printed page. The image datagenerated by the camera(s) can include any suitable number of stillimage(s) and/or video data of the imaged object.

The application 104 can analyze the image data of the document 108 anddetermine one or more data elements 110 that are present in the document108. Such data element(s) 108 can include text portions (e.g.,alphanumeric text) of any suitable length, such as one or morecharacters, words, phrases, sentences, numbers, symbols, barcodes,and/or other text elements. The data element(s) 108 can also include atleast a portion of one or more graphic elements and/or images that areprinted onto the document 108. The document analysis can include anysuitable technique for optical character recognition (OCR), optical wordrecognition (OWR), intelligent character recognition (ICR), intelligentword recognition (IWR), natural language processing (NLP), machinelearning, parsing, and/or other techniques for identifying particulartext elements or other data elements in the document 108.

The data element(s) 110 are communicated, over one or more networks, toone or more server devices 112 executing analysis module(s) 114. Theserver device(s) 112 can include any suitable number and type ofcomputing device(s). In some implementations, the server device(s) 112include distributed computing device(s) (e.g., cloud server(s))providing a cloud computing service. The analysis module(s) 114 analyzethe data element(s) 110 and determine context data 116 that isassociated with the data element(s) 110. The context data 116 may bemetadata and/or descriptive information that provides further contextand/or information regarding the data element(s) 110. In someimplementations, the context data 116 is retrieved from a table, filesystem, database, or other data structure that stores one or moreelements of context data 116 that correspond to each of the dataelements 110. The elements of context data 116 corresponding to each ofthe data elements 110 can be retrieved.

The context data 116 is communicated, over one or more networks, to theuser device 102. The application 104 generates a context overlay 122that includes at least a portion of the context data 116. In someimplementations, the context overlay 122 is arranged such that eachelement of the context data 116 is presented at a position that is nearor otherwise visually corresponds to the data element 110 that isassociated with the context data element. Accordingly, the user viewingthe context overlay 122 that is overlaid on the image of the document108 may readily perceive that each element of the context data 116 isassociated with the corresponding data element 110, providing additionalcontext and/or descriptive information regarding the data element 110.The context overlay 122 can be presented with the document image 120 inthe UI of the application 104 executing on the user device 102.

In the example of FIG. 1, the document 108 being imaged is a financialdocument that describes or references a loan. One of the data elements110 extracted from the document 108 is a loan identifier (ID) such as anumber or other alphanumeric text element that uniquely identifies theloan in the computing system of a financial services organization suchas a bank. On receiving the loan ID, the analysis module(s) 114 canretrieve, from a database of the organization, context data 116regarding the particular loan corresponding to the loan ID. Such contextdata 116 can include information regarding the status of the loan, suchas whether the loan has been approved or denied, whether the loanapplication is currently pending and being processed for approval, orwhether additional information is needed from the user to process theloan. Such context data 116 can be presented in the context overlay 122,in a position within the overlay that approximately corresponds to theposition of the loan ID in the image of the document 108. The contextoverlay 122 can be presented along with the image of the document 108 inthe UI of the application 104, to provide easy-to-access informationregarding the loan identified by the loan ID. Other context data 116 inthis example can include information describing various other dataelement(s) 110 in the document 108, such as helpful explanation of whatcertain terms mean or easy-to-understand description of variousparagraphs or other sections within the document 108.

Other types of documents 108 can also be imaged and presented with acontext overlay 122 that describes portions of the document. Forexample, financial documents can be presented with an overlay thatincludes description and/or definition of various terms and/or othertext portions included in the document.

FIG. 2 depicts an example system for presenting an object image with acontext data overlay, according to implementations of the presentdisclosure. In the example of FIG. 2, another type of physical object202 is being imaged, such as a house or other piece of real property.The object image 208 of the object 202 is captured by the camera(s) 106.The user device 102 executes one or more location module(s) 204 thatdetermine location data 206 describing a current location of the userdevice 102. For example, the location module(s) 204 may receive andanalyze signals from a satellite-based navigation system, such as aversion of the global positioning system (GPS) or other suitable system,to determine the current location of the user device 102. Other suitabletechniques for location determination can also be employed.

The location data 206 can be communicated to the server device(s) 112for analysis by the analysis module(s) 114. In some implementations, thelocation data 206 can include location coordinates (e.g., latitude andlongitude) of the current location, or some other identification of thecurrent location such as a street address. The location data 206 canalso include orientation information describing a direction in which thecamera 106 was pointing when it captured the image(s) of the object,such as the compass direction in which the camera 106 was pointing.

The analysis module(s) 114 may determine the context data 116 based onthe location data 206. For example, in instances where the physicalobject 202 is a piece of real property, the analysis module(s) 114 canaccess a database that lists properties corresponding to variouslocations, and determine the particular piece of real property that isat the location of the user device 102 described in the location data206, and that would be in view of the camera 106 at that location andpointed in the direction described in the location data 206. In someimplementations, the user device 102 may also send the object image 208to the server device(s) 112, and the analysis device(s) 114 may analyzethe image to determine the particular piece of real property in theimage. On determining the property based on the location data and/orimage, the analysis module(s) 114 may retrieve context data 116 thatdescribes the property. For example, the context data 116 may beretrieved from a database listing real estate properties for sale, suchas the multiple listing service (MLS) database or similar database.

At least a portion of the context data 116 may be presented in thecontext overlay 122 atop the object image 208 presented in the UI of theapplication 104. For example, the context overlay 122 may presentinformation regarding the property for sale, such as the asking price,identification number (e.g., MLS number), seller, seller's agent, and/orother information such as number of bedrooms, number of bathrooms,square footage, lot size, school district information, and so forth. Theoverlay may also present information regarding the user's loan status,such as whether the user has a mortgage approved in the amount of theproperty being viewed.

The home buying and mortgage application process is typicallycomplicated, particularly for a first time home buyer who may not knowhow to begin the process and may require considerable assistancethroughout the process. Implementations provide a tool that makes thehome buying process fun and easy to navigate, with a context overlaythat provides an augmented reality view on context data in addition toimage(s) of the properties that a user may be interested in buying.

In some implementations, the user may be authenticated by theapplication and/or the server device(s) 112 prior to receiving access tothe features of the application 104 for imaging objects and viewingcontext overlays. Such authentication can be through credentials such asusername, password, personal identification number (PIN),knowledge-based question and answer (e.g., what is your favorite pet'sname?), and so forth. Authentication may also be through the collectionand verification of biometric data for the user, such as the user'sfingerprint, retinal scan, facial recognition, voice print recognition,heart rate/pulse measurement, neural activity (e.g., brain wave scan),and so forth.

In some implementations, the application may be executing as part of aparticular context, and/or the application may be configured to assistthe user with particular tasks. For example, the application may bedesigned to help the user find a home to purchase or process a loanapplication, and/or the user may have navigated to a section of theapplication that is configured with such a purpose. The application'simage capture and context overlay may be a feature that assists the userwithin this particular context, and that context may be used by theanalysis module(s) 114 to determine the context data 116 to be presentedin the overlay. For example, along with the data elements and/orlocation data sent to the service, the application 104 may also send anindication of what section of the application is being used when theimage is captured (e.g., home purchase help section, loan applicationsection, etc.). That indication may be employed by the analysismodule(s) 114 to determine the appropriate context data 116. Forexample, an image captured while the user is using the home purchasehelp section of the application may be used to infer that the sentlocation data 206 corresponds to the location of a property that theuser is interested in. Based on that inference, the analysis module(s)114 can retrieve context data 116 from the corresponding MLS listing orother real property database, and send the context data 116 to theapplication 104. As another example, an image captured while the user isusing the loan application section of the application may be used toinfer that the sent data elements are extracted from a financialdocument that the user is reading. Based on that inference, the analysismodule(s) 114 can retrieve context data 116 from the corresponding loandatabase or other database, and send the context data 116 to theapplication 104.

In some implementations, the presentation of the context overlay isbased on recognizing text that is present in the document or otherobject being imaged. For example, the user may point their camera (e.g.,of the user device) at the document, and the application can recognitiona loan ID number present in the document. As another example, theapplication can recognize a real property ID number (e.g., MLS listingnumber) present in a flyer advertising the property for sale. Theextracted data elements (e.g., ID number(s)) can be sent to the analysismodule(s) 114, which generate context data 116 that is textualdescription of the item identified by the ID number, and that contextdata 116 is presented in the context overlay 122 atop the image of thedocument or other object.

Other types of context data 116 can also be presented as part of thecontext overlay 122 within the application 104. In some implementations,the context data includes a link, a button, or some other type ofcontrol that is selectable by the user to display additional informationdescribing the imaged object. For example, the overlay can include abutton or link that is presented over or near a data element, such as aloan ID number, in the document. When the button or link is selected,the application can present (in the overlay or elsewhere), the contextdata describing the status or other information regarding the loanreferenced by the loan ID.

In some implementations, the context data presented in the overlayincludes a control that is selectable, by the user, to launch a chatsession between the user some other individual, such as a customerservice representative. The chat session may be a video chat session, anaudio chat session (e.g., telephone call or VOIP call), a text chatsession, and so forth. During the session, the user and other individualmay discuss the document or other object being imaged. For example, aloan officer of a bank may discuss a loan application with the user, ora real estate agent or mortgage specialist may discuss the imagedproperty with the user. In general, the context data in the overlay mayinclude any suitable type and format of information, such as text data,video data, audio data, other image(s), graphics, dynamic selectablecontrols, and so forth. In some implementations, the context data mayinclude pre-recorded snippets of video and/or audio that play inresponse to a command made through the application, such as video oraudio that provides help regarding particular portions of an imageddocument, or regarding some other type of imaged object.

In one example, the application may be used to initiate a mortgageapplication process. For example, the user may point their camera at aproperty for sale, and the application may show the context dataregarding the property. The overlay may also include a control thatenables the user to initiate the mortgage application process, for aloan that is at least the listed purchase price of the property asretrieved by the analysis module(s) 114. The overlay may also include acontrol to launch a real time chat session (e.g., video, audio, and/ortext) with a loan officer or other mortgage specialist to discuss theloan application.

As another example, the user may point their camera at a flyeradvertising a property for sale, and the application 104 may recognizeand retrieve the property ID number (e.g., MLS number) from the flyer.The context data 116 may be generated to include information regardingthe property. As another example, an image of a W2 form, pay stub, orother financial document may be analyzed to retrieve data elements, andthe context data may be presented that is associated with such dataelements. As another example, an image of a business card may beanalysis to retrieve the name or other identifying information from thecard, and the overlay can include information regarding the personidentified by the card as well as a control that is selectable to launcha video or audio introduction from the person, or a real time chatsession between the user and the person.

In some implementations, the presentation of the context overlay isbased on an image that is captured and/or a location of the user device102 when the image is captured. An address or otherwise describedcurrent location of the user device is recognized as being at or near aproperty for sale, and the context data includes retrieved informationregarding that property, to be presented in the overlay with the imageof the property. For example, the overlay can include context data thatis at least a portion of the MLS listing for the property, along with acontrol that is selectable to begin the mortgage application process. Insome instances, the context data is generated through a call from theserver device(s) 112 to a third party service that maintains theappropriate data, such as data describing current properties for sale inan area. In some implementations, the context data 116 presented in theoverlay includes publicly available data that is retrieved from apublished database. The context data can also include video content suchas a virtual tour of the property, a video message from the seller orseller's agent, and so forth. The context data can also include acontrol that enables the user to launch a real time chat session withtheir real estate agent, a loan officer, a customer servicerepresentative, or other individual.

In some implementations, the analysis of the document and/or objectimage is performed by an engine that supports text recognition, imagerecognition, and/or location recognition. The engine may be trained,using any suitable machine learning technique, to generate the contextdata 116 based on input image(s), data element(s), location information,and/or other input data. The analysis may be performed by the analysismodule(s) 114 executing on the server device(s) 112 and/or by theapplication 104 executing on the user device 102.

FIG. 3 depicts a flow diagram of an example process for presenting adocument image with a context data overlay, according to implementationsof the present disclosure. Operations of the process can be performed byone or more of the application 104, the analysis module(s) 114, and/orother software module(s) executing on the user device 104, the serverdevice(s) 112, or elsewhere.

Image(s) of a physical document are received (302), as described abovewith reference to FIG. 1. The image(s) are analyzed (304) to identifydata element(s) present in the document. The data element(s) are sent(306), in a request to the remote service (e.g., the analysis module(s)114). The context data is received in response to the request. Theimage(s) of the physical document are displayed (308) with the contextoverlay that includes the context data.

FIG. 4 depicts a flow diagram of an example process for presenting anobject image with a context data overlay, according to implementationsof the present disclosure. Operations of the process can be performed byone or more of the application 104, the analysis module(s) 114, and/orother software module(s) executing on the user device 104, the serverdevice(s) 112, or elsewhere.

Image(s) of a physical object (e.g., property) are received (402), asdescribed above with reference to FIG. 2. The location data isdetermined (404) that includes a current location and/or orientation(e.g., camera direction) of the user device 102 when the image(s) werecaptured. The location data is sent (406), in a request to the remoteservice (e.g., the analysis module(s) 114). The context data is receivedin response to the request. The image(s) of the physical object aredisplayed (408) with the context overlay that includes the context data.

The context overlay can include any suitable amount of the context datapresented in an appropriate manner to assist the user in understandingthe imaged document or other object. For example, the overlay may be atleast partly transparent, such that the document or object image isvisible under the overlay while the context data is viewable with theoverlay. In some instances, the context overlay may be presented atleast partly to the side of the image(s) in the display.

FIG. 5 depicts an example computing system, according to implementationsof the present disclosure. The system 500 may be used for any of theoperations described with respect to the various implementationsdiscussed herein. For example, the system 500 may be included, at leastin part, in one or more of the user device 102, the server device(s)112, and/or other computing device(s) or system(s) described herein. Thesystem 500 may include one or more processors 510, a memory 520, one ormore storage devices 530, and one or more input/output (I/O) devices 550controllable through one or more I/O interfaces 540. The variouscomponents 510, 520, 530, 540, or 550 may be interconnected through atleast one system bus 560, which may enable the transfer of data betweenthe various modules and components of the system 500.

The processor(s) 510 may be configured to process instructions forexecution within the system 500. The processor(s) 510 may includesingle-threaded processor(s), multi-threaded processor(s), or both. Theprocessor(s) 510 may be configured to process instructions stored in thememory 520 or on the storage device(s) 530. The processor(s) 510 mayinclude hardware-based processor(s) each including one or more cores.The processor(s) 510 may include general purpose processor(s), specialpurpose processor(s), or both.

The memory 520 may store information within the system 500. In someimplementations, the memory 520 includes one or more computer-readablemedia. The memory 520 may include any number of volatile memory units,any number of non-volatile memory units, or both volatile andnon-volatile memory units. The memory 520 may include read-only memory,random access memory, or both. In some examples, the memory 520 may beemployed as active or physical memory by one or more executing softwaremodules.

The storage device(s) 530 may be configured to provide (e.g.,persistent) mass storage for the system 500. In some implementations,the storage device(s) 530 may include one or more computer-readablemedia. For example, the storage device(s) 530 may include a floppy diskdevice, a hard disk device, an optical disk device, or a tape device.The storage device(s) 530 may include read-only memory, random accessmemory, or both. The storage device(s) 530 may include one or more of aninternal hard drive, an external hard drive, or a removable drive.

One or both of the memory 520 or the storage device(s) 530 may includeone or more computer-readable storage media (CRSM). The CRSM may includeone or more of an electronic storage medium, a magnetic storage medium,an optical storage medium, a magneto-optical storage medium, a quantumstorage medium, a mechanical computer storage medium, and so forth. TheCRSM may provide storage of computer-readable instructions describingdata structures, processes, applications, programs, other modules, orother data for the operation of the system 500. In some implementations,the CRSM may be non-transitory and may include a data store thatprovides storage of computer-readable instructions or other informationin a non-transitory format. The CRSM may be incorporated into the system500 or may be external with respect to the system 500. The CRSM mayinclude read-only memory, random access memory, or both. One or moreCRSM suitable for tangibly embodying computer program instructions anddata may include any type of non-volatile memory, including but notlimited to: semiconductor memory devices, such as EPROM, EEPROM, andflash memory devices; magnetic disks such as internal hard disks andremovable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Insome examples, the processor(s) 510 and the memory 520 may besupplemented by, or incorporated into, one or more application-specificintegrated circuits (ASICs).

The system 500 may include one or more I/O devices 550. The I/Odevice(s) 550 may include one or more input devices such as a keyboard,a mouse, a pen, a game controller, a touch input device, an audio inputdevice (e.g., a microphone), a gestural input device, a haptic inputdevice, an image or video capture device (e.g., a camera), or otherdevices. In some examples, the I/O device(s) 550 may also include one ormore output devices such as a display, LED(s), an audio output device(e.g., a speaker), a printer, a haptic output device, and so forth. TheI/O device(s) 550 may be physically incorporated in one or morecomputing devices of the system 500, or may be external with respect toone or more computing devices of the system 500.

The system 500 may include one or more I/O interfaces 540 to enablecomponents or modules of the system 500 to control, interface with, orotherwise communicate with the I/O device(s) 550. The I/O interface(s)540 may enable information to be transferred in or out of the system500, or between components of the system 500, through serialcommunication, parallel communication, or other types of communication.For example, the I/O interface(s) 540 may comply with a version of theRS-232 standard for serial ports, or with a version of the IEEE 1284standard for parallel ports. As another example, the I/O interface(s)540 may be configured to provide a connection over Universal Serial Bus(USB) or Ethernet. In some examples, the I/O interface(s) 540 may beconfigured to provide a serial connection that is compliant with aversion of the IEEE 1394 standard.

The I/O interface(s) 540 may also include one or more network interfacesthat enable communications between computing devices in the system 500,or between the system 500 and other network-connected computing systems.The network interface(s) may include one or more network interfacecontrollers (NICs) or other types of transceiver devices configured tosend and receive communications over one or more networks using anynetwork protocol.

Computing devices of the system 500 may communicate with one another, orwith other computing devices, using one or more networks. Such networksmay include public networks such as the internet, private networks suchas an institutional or personal intranet, or any combination of privateand public networks. The networks may include any type of wired orwireless network, including but not limited to local area networks(LANs), wide area networks (WANs), wireless WANs (WWANs), wireless LANs(WLANs), mobile communications networks (e.g., 3G, 4G, Edge, etc.), andso forth. In some implementations, the communications between computingdevices may be encrypted or otherwise secured. For example,communications may employ one or more public or private cryptographickeys, ciphers, digital certificates, or other credentials supported by asecurity protocol, such as any version of the Secure Sockets Layer (SSL)or the Transport Layer Security (TLS) protocol.

The system 500 may include any number of computing devices of any type.The computing device(s) may include, but are not limited to: a personalcomputer, a smartphone, a tablet computer, a wearable computer, animplanted computer, a mobile gaming device, an electronic book reader,an automotive computer, a desktop computer, a laptop computer, anotebook computer, a game console, a home entertainment device, anetwork computer, a server computer, a mainframe computer, a distributedcomputing device (e.g., a cloud computing device), a microcomputer, asystem on a chip (SoC), a system in a package (SiP), and so forth.Although examples herein may describe computing device(s) as physicaldevice(s), implementations are not so limited. In some examples, acomputing device may include one or more of a virtual computingenvironment, a hypervisor, an emulation, or a virtual machine executingon one or more physical computing devices. In some examples, two or morecomputing devices may include a cluster, cloud, farm, or other groupingof multiple devices that coordinate operations to provide loadbalancing, failover support, parallel processing capabilities, sharedstorage resources, shared networking capabilities, or other aspects.

Implementations and all of the functional operations described in thisspecification may be realized in digital electronic circuitry, or incomputer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them. Implementations may be realized asone or more computer program products, i.e., one or more modules ofcomputer program instructions encoded on a computer readable medium forexecution by, or to control the operation of, data processing apparatus.The computer readable medium may be a machine-readable storage device, amachine-readable storage substrate, a memory device, a composition ofmatter effecting a machine-readable propagated signal, or a combinationof one or more of them. The term “computing system” encompasses allapparatus, devices, and machines for processing data, including by wayof example a programmable processor, a computer, or multiple processorsor computers. The apparatus may include, in addition to hardware, codethat creates an execution environment for the computer program inquestion, e.g., code that constitutes processor firmware, a protocolstack, a database management system, an operating system, or acombination of one or more of them. A propagated signal is anartificially generated signal, e.g., a machine-generated electrical,optical, or electromagnetic signal that is generated to encodeinformation for transmission to suitable receiver apparatus.

A computer program (also known as a program, software, softwareapplication, script, or code) may be written in any appropriate form ofprogramming language, including compiled or interpreted languages, andit may be deployed in any appropriate form, including as a standaloneprogram or as a module, component, subroutine, or other unit suitablefor use in a computing environment. A computer program does notnecessarily correspond to a file in a file system. A program may bestored in a portion of a file that holds other programs or data (e.g.,one or more scripts stored in a markup language document), in a singlefile dedicated to the program in question, or in multiple coordinatedfiles (e.g., files that store one or more modules, sub programs, orportions of code). A computer program may be deployed to be executed onone computer or on multiple computers that are located at one site ordistributed across multiple sites and interconnected by a communicationnetwork.

The processes and logic flows described in this specification may beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows may also be performedby, and apparatus may also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any appropriate kind of digital computer.Generally, a processor may receive instructions and data from a readonly memory or a random access memory or both. Elements of a computercan include a processor for performing instructions and one or morememory devices for storing instructions and data. Generally, a computermay also include, or be operatively coupled to receive data from ortransfer data to, or both, one or more mass storage devices for storingdata, e.g., magnetic, magneto optical disks, or optical disks. However,a computer need not have such devices. Moreover, a computer may beembedded in another device, e.g., a mobile telephone, a personal digitalassistant (PDA), a mobile audio player, a Global Positioning System(GPS) receiver, to name just a few. Computer readable media suitable forstoring computer program instructions and data include all forms ofnon-volatile memory, media and memory devices, including by way ofexample semiconductor memory devices, e.g., EPROM, EEPROM, and flashmemory devices; magnetic disks, e.g., internal hard disks or removabledisks; magneto optical disks; and CD ROM and DVD-ROM disks. Theprocessor and the memory may be supplemented by, or incorporated in,special purpose logic circuitry.

To provide for interaction with a user, implementations may be realizedon a computer having a display device, e.g., a CRT (cathode ray tube) orLCD (liquid crystal display) monitor, for displaying information to theuser and a keyboard and a pointing device, e.g., a mouse or a trackball,by which the user may provide input to the computer. Other kinds ofdevices may be used to provide for interaction with a user as well; forexample, feedback provided to the user may be any appropriate form ofsensory feedback, e.g., visual feedback, auditory feedback, or tactilefeedback; and input from the user may be received in any appropriateform, including acoustic, speech, or tactile input.

Implementations may be realized in a computing system that includes aback end component, e.g., as a data server, or that includes amiddleware component, e.g., an application server, or that includes afront end component, e.g., a client computer having a graphical UI or aweb browser through which a user may interact with an implementation, orany appropriate combination of one or more such back end, middleware, orfront end components. The components of the system may be interconnectedby any appropriate form or medium of digital data communication, e.g., acommunication network. Examples of communication networks include alocal area network (“LAN”) and a wide area network (“WAN”), e.g., theInternet.

The computing system may include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specifics, these should not beconstrued as limitations on the scope of the disclosure or of what maybe claimed, but rather as descriptions of features specific toparticular implementations. Certain features that are described in thisspecification in the context of separate implementations may also beimplemented in combination in a single implementation. Conversely,various features that are described in the context of a singleimplementation may also be implemented in multiple implementationsseparately or in any suitable sub-combination. Moreover, althoughfeatures may be described above as acting in certain combinations andeven initially claimed as such, one or more features from a claimedcombination may in some examples be excised from the combination, andthe claimed combination may be directed to a sub-combination orvariation of a sub-combination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the implementations described above should not beunderstood as requiring such separation in all implementations, and itshould be understood that the described program components and systemsmay generally be integrated together in a single software product orpackaged into multiple software products.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the disclosure. For example, various formsof the flows shown above may be used, with steps re-ordered, added, orremoved. Accordingly, other implementations are within the scope of thefollowing claims.

The invention claimed is:
 1. A computer-implemented method performed bya portable computing device, the method comprising: receiving, by theportable computing device, at least one image of a physical object, theat least one image captured by a camera of the portable computingdevice; determining, by the portable computing device, location dataindicating a geographical location of the portable computing device;communicating, by the portable computing device and over at least onenetwork to at least one server device, the location data and anindication of a particular section of an application that was in use onthe portable computing device at a time that the at least one image wascaptured, wherein the particular section of the application that was inuse on the portable computing device at a time that the at least oneimage was captured is a section of the application that relates to realestate purchases; receiving, from the at least one server device,context data associated with the physical object, the context datacommunicated from the at least one server device over the at least onenetwork and the context data being determined, at least in part, basedon the particular section of the application that was in use on theportable computing device at a time that the at least one image wascaptured; and displaying, by the portable computing device, the at leastone image with an overlay that includes at least a portion of thecontext data, wherein the context data includes data corresponding to aproperty located at the geographical location of the portable computingdevice.
 2. The method of claim 1, wherein the at least one server deviceexecutes a service that determines the context data based on thelocation data and the particular section of the application that was inuse on the portable computing device at a time that the at least oneimage was captured.
 3. The method of claim 1, further comprisingidentifying, within the at least one image, a data element associatedwith the physical object.
 4. The method of claim 3, further comprisingcommunicating, by the portable computing device and over the at leastone network, the data element to the at least one server device, whereinthe at least one server device executes a service that determines thecontext data based on the data element, the location data, and theparticular section of the application that was in use on the portablecomputing device at a time that the at least one image was captured. 5.The method of claim 3, wherein the context data includes a control thatis displayed in the overlay and that is selectable to display additionalinformation describing the data element.
 6. The method of claim 1,wherein the context data includes a control that is displayed in theoverlay and that is selectable to launch a video chat session between auser of the portable computing device and a service representative. 7.The method of claim 3, wherein displaying the at least one image withthe overlay that includes at least a portion of the context datacomprises positioning the overlay proximate to the data element that isassociated with the portion of the context data.
 8. The method of claim3, wherein: the data element is a property identifier; and the contextdata includes information describing a property corresponding to theproperty identifier.
 9. A system, comprising: at least one processor;and a memory communicatively coupled to the at least one processor, thememory storing instructions which, when executed by the at least oneprocessor, cause the at least one processor to perform operationscomprising: receiving at least one image of a physical object, the atleast one image captured by a camera of a portable computing device;determining location data indicating a geographical location of theportable computing device; communicating, over at least one network toat least one server device, the location data and an indication of aparticular section of an application that was in use on the portablecomputing device at a time that the at least one image was captured,wherein the particular section of the application that was in use on theportable computing device at a time that the at least one image wascaptured is a section of the application that relates to real estatepurchases; receiving, from the at least one server device, context dataassociated with the physical object, the context data communicated fromthe at least one server device over the at least one network and thecontext data being determined, at least in part, based on the particularsection of the application that was in use on the portable computingdevice at a time that the at least one image was captured; andproviding, for display, the at least one image with an overlay thatincludes at least a portion of the context data, wherein the contextdata includes data corresponding to a property located at thegeographical location of the portable computing device.
 10. The systemof claim 9, wherein the at least one server device executes a servicethat determines the context data based on the location data and theparticular section of the application that was in use on the portablecomputing device at a time that the at least one image was captured. 11.The system of claim 9, further comprising identifying, within the atleast one image, a data element associated with the physical object. 12.The system of claim 11, further comprising communicating, by theportable computing device and over the at least one network, the dataelement to the at least one server device, wherein the at least oneserver device executes a service that determines the context data basedon the data element, the location data, and the particular section ofthe application that was in use on the portable computing device at atime that the at least one image was captured.
 13. The system of claim11, wherein the context data includes a control that is displayed in theoverlay and that is selectable to display additional informationdescribing the data element.
 14. The system of claim 9, wherein thecontext data includes a control that is displayed in the overlay andthat is selectable to launch a video chat session between a user of theportable computing device and a service representative.
 15. The systemof claim 11, wherein displaying the at least one image with the overlaythat includes at least a portion of the context data comprisespositioning the overlay proximate to the data element that is associatedwith the portion of the context data.
 16. The system of claim 11,wherein: the data element is a property identifier; and the context dataincludes information describing a property corresponding to the propertyidentifier.
 17. One or more non-transitory computer-readable mediastoring instructions which, when executed by at least one processor,cause the at least one processor to perform operations comprising:receiving at least one image of a physical object, the at least oneimage captured by a camera of a portable computing device; determininglocation data indicating a geographical location of the portablecomputing device; communicating, over at least one network to at leastone server device, the location data and an indication of a particularsection of an application that was in use on the portable computingdevice at a time that the at least one image was captured, wherein theparticular section of the application that was in use on the portablecomputing device at a time that the at least one image was captured is asection of the application that relates to real estate purchases;receiving, from the at least one server device, context data associatedwith the physical object, the context data communicated from the atleast one server device over the at least one network and the contextdata being determined, at least in part, based on the particular sectionof the application that was in use on the portable computing device at atime that the at least one image was captured; and providing, fordisplay, the at least one image with an overlay that includes at least aportion of the context data, wherein the context data includes datacorresponding to a property located at the geographical location of theportable computing device.
 18. The one or more computer-readable mediaof claim 17, wherein the at least one server device executes a servicethat determines the context data based on the location data and theparticular section of the application that was in use on the portablecomputing device at a time that the at least one image was captured. 19.The one or more computer-readable media of claim 17, further comprisingidentifying, within the at least one image, a data element associatedwith the physical object.
 20. The one or more computer-readable media ofclaim 19, further comprising communicating, by the portable computingdevice and over the at least one network, the data element to the atleast one server device, wherein the at least one server device executesa service that determines the context data based on the data element,the location data, and the particular section of the application thatwas in use on the portable computing device at a time that the at leastone image was captured.